Why Speaker Time Alignment Doesn't Work